System and method for parallel image processing and routing

ABSTRACT

Various embodiments relate to systems and methods for simultaneously switching input image streams to output devices, while providing optional image processing functions on the image streams. Certain embodiments may enable multiple users/viewers to collaboratively control such systems and methods. Additionally, some embodiments may enable control by a set of computer input devices (e.g., keyboard and mouse) to switch between multiple computer systems, possibly by following the movement of a computer input device cursor, between virtual displays, as the cursor is controlled by the set of computer input devices.

PRIORITY CLAIMS

This application is a continuation of U.S. patent application Ser. No.15/170,575 filed Jun. 1, 2016, which claims priority from and is acontinuation-in-part of U.S. patent application Ser. No. 13/430,489,filed Mar. 26, 2012, which is a continuation-in-part of U.S. patentapplication Ser. No. 12/776,048, filed May 7, 2010, which claims thebenefit of U.S. Provisional Patent Application Ser. No. 61/182,624,filed May 29, 2009, and 61/234,577, filed Aug. 17, 2009. Each of theforegoing applications is hereby incorporated by reference.

TECHNICAL FIELD

The present invention relates generally to video and data streamprocessing, and more particularly, some embodiments relate tosimultaneously processing multiple image streams with selectablealgorithms in real time.

DESCRIPTION OF THE RELATED ART

Unmanned aerial vehicles (UAVs) are often equipped with a variety ofsensors that provide a wide range of imagery and telemetry data to aground control station. UAVs are operated under a broad range ofconditions and are equipped with various sensor systems to allow thisoperation. For example, a UAV may be equipped with a video camera, aninfrared camera, and a synthetic aperture radar system, allowing it tooperate under a broad range of conditions. Data from these differentsources are typically displayed on different display devices within theground control station. These data streams are typically transmittedusing a variety of protocols and formats according to the source deviceand purpose of the data stream. Accordingly, a ground control stationtypically devotes separate resources to the separate data streams.

To display visual information from data streams (e.g., comprisingimagery, telemetry, tactical or technology information), large panelvideo and multi-screen displays are commonly employed in such contextsas Network Operations Centers (NOCs), Tactical Operations Centers(TOCs), Emergency Operations Centers (EOCs), security, entertainment,operating rooms, theaters, industrial design and retail venues. Suchdisplays are often referred to as video walls, info walls, data walls,tiled displays, virtual walls, display walls, or virtual monitor walls.Large panel video and multi-screen displays provide groups of peoplewith visual information that can be automatically updated, or arefrequently used for enabling collaboration among viewers. Where a largepanel video and multi-screen display is used for group collaboration,there is generally a requirement that the group has the ability toupdate and reconfigure the visual information displayed, which isusually facilitated through a video switch. Traditional video switchesare controlled through a switch box, a keyboard, or a local connection(via an RS-232 port or Ethernet port) and have only a single point forcontrol access.

In some contexts, visual data streams to a single large panel videodisplay or a multi-screen display configuration are provided by two ormore computer systems, each being controlled by a computer operator(i.e., user) using such input/output (I/O) devices as keyboards, mice,and as video monitor. At times, it is convenient for the computeroperator to share the 10 devices between the computers by means of adevice that switches the 10 devices between the multiple computers.These switches, often referred to as a Keyboard-Video-Mouse (KVM)switch, are commanded by the computer operator (e.g., commanded by aspecial keyboard sequence or a button on the KVM switch) to switch acommon keyboard, mouse, or video monitor between controlling thecomputer systems.

BRIEF SUMMARY OF THE EMBODIMENTS OF THE INVENTION

The present invention is directed toward a system and method forsimultaneously switching multiple input image streams to multiple outputdevices while providing optional image processing functions on the imagestreams. The incoming image streams may come from computers, cameras,radar, medical devices, or other image stream generating devices. Theincoming streams may vary in resolution, frame rate, format, andprotocol according to the characteristics and purpose of theirrespective source devices. The incoming image streams can be iterativelyrouted through a reconfigurable array of image processing circuit cardsto operate on one or more streams using preprogrammed algorithms. Aprocessed stream can be routed through another processing function, orto an output circuit card that sends the completed image to the externaldisplay, recording device, across a network, or to another logicaldestination.

For example, a system or method in accordance with the present inventionmay selectively and dynamically route one or more image input streamsthrough one or more dynamically selectable and iterative algorithmprocessing elements (or modules) before the resulting processed streamsare routed to one or more selectable outputs. For some embodiments,dynamic selectability with respect to inputs means that one or moreinputs can be selected or unselected in real time for routing to one ormore image processing elements. For additional embodiments, dynamic anditerative selectability with respect to processing elements means that aselected image stream can be routed to one or more image processingelements simultaneously and in real time. The routing may be based uponcriteria relating to the image stream's source, the image stream'scontent, or on the processing results of a preceding image processingelement. The output of an image processing element may be directed backto the system or method for routing to a subsequent image processingelement, or to one or more image outputs that supply an image stream toan output device (e.g., display, image recorder, or other imageprocessor, or transmission device).

An exemplary system for collaboratively switching an image stream inputto an image stream output, comprising: an image stream input interface;an image stream output interface; a first image processing module,wherein the first image processing module is configured to accept animage stream from the image stream input interface or another imageprocessing module, apply an image processing function to the imagestream, and output a processed image stream; and a switching matrix,wherein the switching matrix is in communication with the image streaminput interface, the image stream output interface, and the first imageprocessing module such that the switching matrix can selectively map(i.e., route) the image stream input interface to the image streamoutput interface or to the first image processing module, and theswitching matrix can selectively map the processed image stream from thefirst image processing module to the image stream output interface or toa second image processing module. An image stream includes both a steadyflow of image frames (i.e., video) and image stills captured at a setinterval (e.g., once every hour). Additionally, for some embodiments thesystem comprises a plurality of image processing modules, a plurality ofimage stream input interfaces, or a plurality of image stream outputinterfaces. Furthermore, in further embodiments, the system may beintegrated into a display having a display output, wherein the imagestream output interface is in communication with the display output.

In some embodiments, the image processing module is a field programmablegate array (FPGA) to which algorithms, such as image processingfunctions, can be programmed and changed if desired. Additionally, withthe use of multiple images processing functions, algorithms, such asimage processing functions, can be executed in a predetermined oradaptive sequence. Exemplary algorithms include convolutions, real-timeadaptive histograms, and algorithms capable of processing multipleframes simultaneously (e.g., image subtraction function).

After reading this description, one of ordinary skill in the art wouldunderstand that, depending on the embodiment, either the image streaminput interface, the image stream output interface, or both may utilizeunidirectional communication or bidirectional communication with inputdevices and output devices. For example, a system may be configured toreceive control information from a controller interface device via animage stream output interface, or to send control information to animage source via an image stream input interface. In another example,the control information is received by the system through the DisplayData Channel (DDC). Depending on the embodiment, the system may beconfigured to send control information to a device external to thesystem, through the image stream input interface.

In some embodiments, the switching matrix may selectively map an imagestream input interface or a processed image stream in real-time. Infurther embodiments, the switching matrix may selectively map the imagestream input interface to more than one image processing module or tomore than one image stream output interface simultaneously.Additionally, in some embodiments, the switching matrix may selectivelymap the processed image stream to more than one image processing moduleor to more than one image stream output interface simultaneously. Inother embodiments, the switching matrix may selectively map an imagestream input interface or the processed image stream based on acriterion. For example, the switching matrix may selectively map animage stream input interface or a processed image stream based on itssource or content. In another example, the switching matrix mayselectively map a processed image stream based on the results of apreceding image processing module. Depending on the embodiment, theimage processing module may have the capability of processing aplurality of image streams in parallel.

The image stream interface for some embodiments may be configured toreceive the image stream from an image stream capture device, an imagestream playback device, a computer system, a sensor device (e.g.,sensors typically found on an aircraft, such as a radar system), or amedical device (e.g., endoscope). The image stream output interface forsome embodiments may be configured to output to a display (e.g., liquidcrystal display monitor), a computer system, or recording device (e.g.,digital video recorder). Further, in some embodiments, the system may beconfigured to output an image stream through a virtual display.

In numerous embodiments, the system further comprises a data inputinterface, wherein the switching matrix is further in communication withthe data input interface such that the switching matrix can furtherselectively map the data input interface to the image stream outputinterface or to the first image processing module. For some suchembodiments, the image stream input interface may comprise the datainput interface.

With respect to protocols and interface types, in various embodiments,the image stream input interface or image stream output interface has aDigital Video Interface (DVI) connector, High-definition MultimediaInterface (HDMI) connector, a Bayonet Neill-Concelman (BNC) connector, afiber optic connector, a DisplayPort connector, a Universal Serial Bus(USB) connector, or a Firewire (IEEE1394) connector. In regard toformats, the image stream input interface is configured to convert fromor the image stream output interface is configured to: a Digital VideoInterface (DVI) standard (e.g., DVI-D digital mode, DVI-A analog mode),a High-definition Multimedia Interface (HDMI) compatible standard, aRed-Green-Blue (RGB) Analog standard, a Red-Green-Blue (RGB) Digitalstandard, a DisplayPort standard, a National Television System Committee(NTSC) standard, a Phase Alternate Line (PAL) standard, or a SerialDigital Interface (SDI) standard.

The image processing function in some embodiments may comprise an imagestream mix function, an image stream scale function, an image streamblend function, an image stream encoding algorithm, or an image streamenhancement function. For those embodiments having an image streamenhancement function, the image stream enhancement function may comprisea de-haze function, a de-blur function, a shadow function, a dawn-duskfunction, a fusion function, a (motion) stabilization function, athermal turbulence function, an equalization function, an edge detectionfunction, a rain and fog function, or a light optimizing function.Further, in some embodiments, the system can eliminate or reduce framelatency by configuring an image processing module to apply processingresults of a given frame to a subsequent frame.

In additional embodiments, where the image stream is provided by anendoscope, the image processing function can enhance the image streamfor detecting texture differences in living tissue, detecting a polyp,detecting anomalous tissue, detect blood circulation, or identifyingreference locations for subsequent registration of images from anothermodality. In further embodiments, a light optimizing function may adjusta color channel for the image stream in order to detect polyps. Forexample, the system may apply image processing functions in real time toimage feeds from an endoscope in order to detect differences in livingtissue, detect anomalous tissue, determine boundaries where tissuetexture changes (for tissue volume and size information), and identifyreference locations for subsequent registration of images from othermodalities.

With respect to the last objective for image processing function, imageregistration is the process of determining the alignment between twoimages that have overlapping regions, where the images may be acquiredat different times and/or by different sensors. The difficulty inaligning such images is further increased during multi-modal imageregistration, where the images are acquired using different imagingtechniques (e.g., two images that share similar content may have verydifferent intensity mappings).

One area where the alignment of images of different modalities (i.e.,multi-modal image registration) is important is in the field of medicalimaging. Different medical imaging techniques, such as x-rays, computedtomography (CT), magnetic resonance imaging (MRI), and positron emissiontomography (PET), reveal different types of information about the bodythat can be used in the diagnosis of a disease. For example, considerthe comparison of a slice from a cranial T1-weighted MRI scan and itscorresponding CT scan. The bone structure of the head, for example, willbe clearly visible in the CT scan, and indicated by the bright areas inthe image. The bone structure has a higher radiodensity than thesurrounding tissue. This is not the case for an MRI scan where, unlikethe CT scan, tissue differences can be clearly distinguished.Accordingly, the alignment of images of different modalities (e.g.,image from a MRI scan and a CT scan) would allow for a better overallpicture of a patient's condition. Accordingly, in some embodiments,algorithms such as edge detection, sum of squared differences (SSD) andFast Fourier Transform (FFT) are programmed into the image processingmodule such that multi-modal registration of images, such as frommultiple medical imaging modalities, can be achieved. Sum of squareddifferences (SSD) is a common metric used for judging image similarityby a weighted sum of squared differences (SSD) cost function. Theevaluation of the SSD cost function can be performed efficiently usingthe Fast Fourier Transform (FFT) to determine the optimal translationbetween two images based on pixel intensities.

In some embodiments, the image stream enhancement function performed bythe image processing module comprises an auto-focus function capable ofadjusting focus of an image stream capture device that is coupled to theimage stream input interface and producing the image stream. Forexample, the auto-focus function may comprise the operations of:performing an edge detection algorithm on the image stream, therebyproducing the processed image stream; and generating focus instructionsfor the image stream capture device based on the processed image stream.In further such embodiments, the operations of the auto-focus functionare repeated iteratively until the image stream capture device isin-focus. In other embodiments, the edge enhancement algorithm utilizesa Laplacian convolution.

In certain embodiments, the system further comprises a control interfacethrough which a user may operate or configure the system, among otherthings (e.g., check status of the system). For some such embodiments,the control interface may comprise a display touch panel having agraphical user interface, a keyboard, or a mouse, which are all examplesof human machine interfaces (HMIs). For particular embodiments where theimage stream output interface is outputting to a computer system, thecontrol interface may be directly connected to the system and notconnected to the computer system.

In some embodiments, the system can detect what type of image streaminput interface, image stream output interface, or image processingmodule is present in the system, and configure the system according tothe type. Further, in some embodiments, the system can detect what typeof image stream input interface, image stream output interface, or imageprocessing module is present in the system, and configure the controlinterface according to the type. For example, the system may detect thetypes of circuit cards present in each card slot of its chassis,self-configure the mix of image input cards, image processing cards, andimage output cards detected, and present to the user a visualrepresentation interface according to the mix for the purpose ofproviding image routing choices. It should be noted that in someembodiments the control interface comprises virtual buttons.

Additional embodiments provide for methods for switching an image streaminput to an image stream output. In particular, one such methodcomprises: receiving an image stream through an image stream inputinterface; using a switching matrix to selectively map the image streamto: a first image processing module, wherein the first image processingmodule is configured to accept the image stream from the image streaminput interface or another image processing module, apply an imageprocessing function to the image stream, and output a processed imagestream, or an image stream output interface. For some such embodiments,if the switching matrix selectively maps the image stream to the firstimage processing module, the first image processing module may performthe operations of: receiving the image stream from the image streaminput interface or another image processing module; applying the imageprocessing function to the image stream; and outputting the processedimage stream. Furthermore, for some such embodiments, if the switchingmatrix selectively maps the image stream to the first image processingmodule, the switching matrix may selectively map the processed imagestream to a second image processing module or to the image stream outputinterface.

Various embodiments may enable multiple users/viewers to collaborativelycontrol a system or method for switching multiple input image streams tomultiple output devices, while providing optional image processingfunctions on the image streams. The collaborative control may, in realtime, to create, remove, reposition, or resize one or more virtualdisplays being outputted to an output device. Virtual displays maypermit a collaborative user/viewer to split-screen, implement apicture-in-picture (PIP) window, and determine image stream source for avirtual display (e.g., to display new information). The collaborativecontrol may allow multiple points of access, whereby multiplecollaborative users/viewers can simultaneously access the collaborativecontrol and dynamically update or reconfigure the virtual displays orthe image stream being displayed in a virtual display.

Certain embodiments may enable control by a set of computer inputdevices (e.g., keyboard and mouse) to switch between multiple computersystems using switching systems and methods described herein. Switchingthe computer input device between the computer systems may befacilitated by following the computer input device cursor insteadbetween virtual displays.

An exemplary system for collaboratively switching an image stream inputto an image stream output, comprising: a plurality of image stream inputinterfaces; an image stream output interface; a switching matrix incommunication with the plurality of image stream input interfaces, andthe image stream output interface, wherein the switching matrix isconfigured to selectively map an image stream input interface of theplurality of image stream input interfaces to the image stream outputinterface; and a collaborative control interface through which two ormore users can operate the system, configure the system (e.g., configurea selectively mapping by the switching matrix), or check status of thesystem, wherein the collaborative control interface is configured to besimultaneously accessed by at least two of the users.

The system for collaboratively switching an image stream input to animage stream output, may comprise: a first image processing moduleconfigured to accept an image stream from the image stream inputinterface of the plurality of image stream input interfaces or anotherimage processing module, apply an image processing function to the imagestream, and output a processed image stream; and a second imageprocessing module, wherein the switching matrix is in furthercommunication with the first image processing module and the secondimage processing module. The switching matrix may be further configuredto: selectively map the image stream input interface of the plurality ofimage stream input interfaces to the image stream output interface or tothe first image processing module, and selectively map the processedimage stream from the first image processing module to the image streamoutput interface or to a second image processing module.

The collaborative control interface may comprise a web-based interfacethat enables operation of the system, configuration of the system, or astatus check of the system, and wherein the web-based interface isaccessible by the users by way of a network connection with the system(e.g., Ethernet, WiFi, or serial connection). The collaborative controlinterface may be further configured to receive commands from the users,store the commands into a queue, and perform the commands on the systemin accordance with the queue. The commands may relate to operating thesystem, configuring the system, or checking a status of the system. Thesystem may detect what type of image stream input interface, imagestream output interface, or image processing module is present in thesystem, and configure the collaborative control interface according tothe type.

Where at least one of the users is an administrative user and at leastone of the users is a non-administrative user, a command from theadministrative user through the collaborative control interface may beperformed before a command from the non-administrative user. Where atleast one of the users is an administrative user and at least one of theusers is a non-administrative user, the administrative user may operatethe system, configure the system, or check a status of the systemthrough the collaborative control interface at the exclusion of thenon-administrative user.

The system may be configured to output an image stream to a virtualdisplay, in a mapped image stream, being outputted to the image streamoutput interface. Accordingly, configuring the system may includecreating the virtual display, closing the virtual display, resizing thevirtual display, repositioning the virtual display within the mappedimage stream being outputted to the image stream output interface,moving the virtual display to another mapped image stream beingoutputted to another image stream output interface (e.g., move a virtualdisplay from a display one to a display two), or repositioning thevirtual display within the mapped image stream being outputted toanother image stream output interface.

For some embodiments, the switching matrix may selectively map two ormore image stream input interfaces of the plurality of image streaminput interfaces to the image stream output interface simultaneously.Additionally, the switching matrix may selectively map image streaminput interface of the plurality of image stream input interfaces to twoor more image stream output interfaces simultaneously.

An exemplary method for collaboratively switching an image stream inputto an image stream output, comprising: receiving a first command from afirst user by way of a collaborative control interface accessible by twoor more users simultaneously, wherein the first command relates toconfiguring a switching matrix in communication with a plurality ofimage stream input interfaces and an image stream output interface, andwherein the switching matrix is configured to selectively map an imagestream input interface of the plurality of image stream input interfacesto the image stream output interface; and receiving a second commandfrom a second user by way of the collaborative control interface,wherein the second command relates to configuring the switching matrix.The method may then store the first command and the second command(e.g., based on time of receipt, command type, user or user type issuingthe command), in a queue; and perform the first command and the secondcommand in accordance with the queue, thereby configuring the switchingmatrix in accordance with the first command and the second command.Subsequently, the method may receive an original image stream from theimage stream input interface of the plurality of image stream inputinterfaces; and using the switching matrix (e.g., as configured by thefirst and second commands) to selectively map the original image streamto the image stream output interface.

For some embodiments, using the switching matrix to selectively map theoriginal image stream to the image stream output interface may compriseusing the switching matrix to selectively map the original image streamto an image processing module, wherein the image processing module isconfigured to accept the original image stream from the image streaminput interface of the plurality of image stream input interfaces, applyan image processing function to the original image stream, and output aprocessed image stream. Using the switching matrix to selectively mapthe original image stream to the image stream output interface mayfurther comprise using the switching matrix to selectively map theprocessed image stream to the image stream output interface or anotherimage processing module.

In various embodiments, storing the first command and the second commandin the queue may be based on an order in which the first command and thesecond command are received, based on a first priority of the firstcommand and a second priority of the second command, or based on a firstuser type of the first user and a second user type of the second user.

Where the switching matrix outputs the original image stream to avirtual display, in a mapped image stream, being outputted to the imagestream output interface, the first command or the second command mayinclude creating the virtual display, closing the virtual display,resizing the virtual display, repositioning the virtual display withinthe mapped image stream being outputted to the image stream outputinterface, or moving the virtual display to another mapped image streambeing outputted to another image stream output interface.

An exemplary system for switching control between a plurality ofcomputer systems, comprising a plurality of image stream inputinterfaces, wherein a first image stream input interface of theplurality of image stream input interfaces is configured to couple witha first computer system of the plurality of computer systems, andwherein a second image stream input interface of the plurality of imagestream input interfaces is configured to couple with a second computersystem of the plurality of computer systems. The system may furthercomprise a plurality of computer input device interfaces (e.g.,Universal Serial Bus [USB], PS/2, AT connector, Bluetooth, Infrared[IF], or FireWire), wherein a first computer input device interface ofthe general plurality of computer input device interfaces is configuredto couple with the first computer system, and wherein a second computerinput device interface of the plurality of general computer input deviceinterfaces is configured to couple with the second computer system. Thesystem may additionally comprise: a central computer input deviceinterface (e.g., Universal Serial Bus [USB], PS/2, AT connector,Bluetooth, Infrared [IF], or FireWire); a central image stream outputinterface; and a switching matrix in communication with the plurality ofimage stream input interfaces, the plurality of general computer inputdevice interfaces, the central computer input device interface, and thecentral image stream output interface. The switching matrix may beconfigured to map the first image stream input interface to a firstvirtual display being outputted through the central image stream outputinterface and map the second image stream input interface to a secondvirtual display being outputted through the central image stream outputinterface. The switching matrix may be further configured to selectivelymap the central computer input device interface to the first computerinput interface or the second computer input interface based on alocation of an input cursor with respect to a first display region beingoutputted through the central image stream output interface or a seconddisplay region being outputted through the central image stream outputinterface. The first virtual display may be in the first display region,the second virtual display may be the second display region, and theinput cursor may be controlled by way of the central computer inputdevice interface.

Selectively mapping the central computer input device interface to thefirst computer input interface or the second computer input interfacemay be further based on whether the input cursor is entering or exitingthe first display region or the second display region. In someembodiments, the first virtual display may coincide with the firstdisplay region, and the second virtual display may coincide with thesecond display region. Depending on the embodiment, the input cursor maybe a mouse being controlled by a mouse coupled to the central computerinput device interface or a keyboard cursor being controlled by akeyboard coupled to the central computer input device interface.

An exemplary method for switching control between a plurality ofcomputer systems, comprising: receiving a first image stream at a firstimage stream input interface configured to couple with a first computersystem of the plurality of computer systems; and receiving a secondimage stream at a second image stream input interface configured tocouple with a second computer system of the plurality of computersystems. The method may further comprise: using a switching matrix tomap the first image stream from the first image stream input interfaceto a first virtual display being outputted through a central imagestream output interface; and using the switching matrix to map thesecond image stream from the second image stream input interface to asecond virtual display being outputted through the central image streamoutput interface. The method may additionally comprise receiving inputinformation from a central computer input device interface; and usingthe switching matrix to selectively map the control information from thecentral computer input device interface to a first computer input deviceinterface or a second computer input device interface based on alocation of an input cursor with respect to a first display region or asecond display region. The first display region may be outputted throughthe central image stream output interface, and the second display regionmay be outputted through the central image stream output interface. Thefirst computer input device interface may be configured to couple withthe first computer system, and the second computer input deviceinterface may be configured to couple with the second computer system.The first virtual display may be in the first display region, the secondvirtual display may be the second display region, and the input cursormay be controlled by way of the central computer input device interface.

Other embodiments provide for a computer readable storage medium havinginstructions embedded thereon to cause a processor to perform operationssimilar to those described above with respect to the various systems andmethods in accordance with the present invention.

Other features and aspects of the invention will become apparent fromthe following detailed description, taken in conjunction with theaccompanying drawings, which illustrate, by way of example, the featuresin accordance with embodiments of the invention. The summary is notintended to limit the scope of the invention, which is defined solely bythe claims attached hereto.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention, in accordance with one or more variousembodiments, is described in detail with reference to the followingfigures. The drawings are provided for purposes of illustration only andmerely depict typical or exemplary embodiments of the invention. Thesedrawings are provided to facilitate the reader's understanding of theinvention and shall not be considered limiting of the breadth, scope, orapplicability of the invention. It should be noted that for clarity andease of illustration these drawings are not necessarily made to scale.

FIG. 1A illustrates an exemplary environment in which an embodiment ofthe invention might be employed.

FIG. 1B illustrates an exemplary system process of an embodiment of theinvention.

FIG. 2A illustrates an exemplary configurable system for simultaneouslyprocessing multiple data streams according to an embodiment of theinvention.

FIG. 2B illustrates an exemplary video input module according to anembodiment of the invention.

FIG. 2C illustrates an exemplary video output module according to anembodiment of the invention.

FIG. 2D illustrates an exemplary functional image processing moduleaccording to an embodiment of the invention.

FIG. 2E illustrates an exemplary method of image processing performed bya functional image processing module according to an embodiment of theinvention.

FIG. 3A illustrates an example of iterative image processing accordingto an embodiment of the invention.

FIG. 3B illustrates an exemplary switch fabric implementation ofiterative image processing according to a particular embodiment of theinvention.

FIG. 3C illustrates an exemplary method of coupling switch fabricsaccording to an embodiment of the invention.

FIG. 4 illustrates an exemplary initial view of a GUI of anon-configured system according to an embodiment of the invention.

FIG. 5 illustrates an exemplary first step of the configuration of asystem according to an embodiment of the invention.

FIG. 6 illustrates an exemplary second step of the configuration of asystem according to an embodiment of the invention.

FIG. 7 illustrates an exemplary third step of the configuration of asystem according to an embodiment of the invention.

FIG. 8 illustrates an exemplary fourth step of the configuration of asystem according to an embodiment of the invention.

FIG. 9A illustrates how system and connected device control can becentralized or distributed using exemplary system routing functionsaccording to an embodiment of the invention.

FIG. 9B illustrates an exemplary switch fabric implementation of devicecontrol according to an embodiment of the invention.

FIG. 9C illustrates an alternative configuration of an exemplary switchfabric chassis according to an embodiment of the invention.

FIG. 10 illustrates an environment in which an exemplary system iscollaboratively switching between an image stream input interface and animage stream output interface according to an embodiment of theinvention.

FIG. 11 illustrates an exemplary web-based collaborative controlinterface in use according to an embodiment of the invention

FIG. 12 illustrates output from an exemplary system for collaborativelyswitching according to an embodiment of the invention.

FIG. 13 illustrates an exemplary method for switching control between aplurality of computer systems according to an embodiment of theinvention.

FIG. 14 illustrates an exemplary system for switching control between aplurality of computer systems according to an embodiment of theinvention.

FIG. 15 illustrates an exemplary control interface for switching controlbetween a plurality of computer systems according to an embodiment ofthe invention.

FIG. 16 illustrates exemplary image stream interfaces and computer inputdevice interfaces of an exemplary system according to an embodiment ofthe invention.

FIG. 17 illustrates an exemplary method for switching control between aplurality of computer systems according to an embodiment of theinvention.

FIG. 18 illustrates an exemplary computing module with which variousfeatures of embodiments of the invention may be implemented.

The figures are not intended to be exhaustive or to limit the inventionto the precise form disclosed. It should be understood that theinvention can be practiced with modification and alteration, and thatthe invention be limited only by the claims and the equivalents thereof.

DETAILED DESCRIPTION OF THE EMBODIMENTS OF THE INVENTION

Before describing the invention in detail, it is useful to describe anexemplary environment with which the invention can be implemented. Onesuch example is that of a military operation, an example of which isshown in FIG. 1A. Video feeds and other data are processed and relayedby a system 54 to one or more destinations 55 and 56. For example, thevideo feeds and other data may comprise satellite imagery feedstransmitted by satellite 50; airborne sources of information such aslive video or radar feeds from aircraft, such as unmanned aerialvehicles (UAVs) 52 or a manned surveillance aircraft (not shown);ground-based information, such as imagery from ground vehicle 53; or anyother video or data feed, such as prerecorded video of terrain stored incomputers 51 for use in providing contextual information for a livevideo stream. In this environment, the data from the various datasources are combined in system 54 and relayed to destinations 55 and 56.For example, destination 55 might comprise a headquarters that isoverseeing the military operation and destination 56 may comprise amobile operations center in charge of coordinating the operation or aportion of the operation. For example, mobile operations center 56 maycomprise a UAV operations center charged with piloting the UAVs 52. Inmany implementations, destinations 55 and 56 may contain a plurality ofdifferent devices to receive incoming data, such as multiple displays,storage devices, or satellite transmitters. Moreover, various personnelmay desire access to all or a portion of the available data.Accordingly, system 54 may be configured to selectively transmitdifferent portions of the data to the various displays according to theneeds of the personnel.

FIG. 1B illustrates an exemplary system process of an embodiment of theinvention deployed in the environment of FIG. 1A. A plurality of inputdevices, such as computers 62 and 61, storage devices 60, and sensordevices 58 and 59 transmit a corresponding plurality of image feeds tothe system 67. For example, the image feeds might comprise satelliteimagery 63, high-altitude terrain imagery 64, prerecorded terrainimagery 65, or an infrared video feed 66. The image feeds are thentransmitted to the system 67 for processing and routing to a variety ofoutput devices. A matrix switch 68 allows the image feeds to be directlyrouted to the output devices or to be routed through one or more imageprocessing or other functional modules. For example, a system mightcomprise a dedicated subsystem of image processing modules 69 to performmixing, scaling, blending functions, and image enhancement functions. Insome embodiments, the most common image processing functions may beperformed on a dedicated subsystem 69. In the embodiment illustrated inFIG. 1B, these are mixing multiple image streams together, for exampleto create a virtual display 74; scaling image streams, for exampleinterpolating an image to increase image resolution for comfortableviewing on a larger screen; and blending image streams, for example tooverlay one image stream on top of another. In these embodiments,configurable or interchangeable functions may be performed on a secondsubsystem 70. Such functions might comprise image enhancement functionsas image stabilization, image de-blurring, image de-hazing, dawn/duskcompensation, temporal blending, image or sensor fusion algorithms, oredge detection, or such functions as video encoding or compressionalgorithms such as an H.264 encoding algorithm. The processed imagestreams may be routed to an output device, or rerouted through anotherfunctional module. For example, a composite image stream could be formedby performing a sensor fusion algorithm on an infrared video stream andan optical video stream, then the composite image stream could be routedthrough an encoding module to form an encoded composite of image streamsfor storage on a video storage device. A variety of different outputdevices may be used with embodiments of the invention, such as videodisplays 73, virtual display 74 comprising one or more mixed or overlaidimage streams, or video storage device 71.

Further image enhancement functions can also include an auto-focusfunction configured to adjust the focus of an input device coupled tothe system 67 based on the image feed provided by the input device. Forexample, according to one embodiment, the auto-focus function within animage processing module 69 could adjust the focus of an input device byfirst receiving an image feed from the input device, performing an edgedetection algorithm on the image feed, and then producing a processedimage feed that would enable easier detection of an edge. Next, based onthe processed image feed, the auto-focus function would generate a focusinstruction for the input device. Once the focus instructions have beenimplemented by the input device, the process may iteratively repeatuntil the input device comes into focus. In other words, the new imagefeed provided by the now adjusted input device would be provided to theauto-focus function, the auto-focus function would perform edgedetection on the new image feed, and the auto-focus function would, ifnecessary, generate additional focus instructions for the input deviceto implement based on the new processed image feed. Optionally, theauto-focus function may utilize a Laplacian convolution in its edgedetection algorithm.

From time-to-time, the present invention is described herein in terms ofthis exemplary environment. Description in terms of this environment isprovided to allow the various features and embodiments of the inventionto be portrayed in the context of an exemplary application. Afterreading this description, it will become apparent to one of ordinaryskill in the art how the invention can be implemented in different andalternative environments.

FIG. 2A illustrates an exemplary configurable system for simultaneouslyprocessing multiple data streams according to an embodiment of theinvention. A plurality of data sources 99 are coupled to the system 98by way of a plurality of input modules 100. The plurality of datasources 99 transmit a corresponding plurality of data streams to thesystem 98. Depending on the type of data source, these data streams mayvary in resolution, frame rate, format, and protocol. For example, ahelmet mounted camera may transmit a low resolution and low frame rateimage stream to the system 98 that is both compressed and encoded; aUAV-mounted camera may provide an encoded high resolution and high framerate image stream; and a computer may provide a static or slowly varyingchroma mask for inserting information into a displayed image stream.Accordingly, in this embodiment, the input modules 100 are configured toreceive these different data streams and convert them to a common formatfor use in system 98. For example, inputs 100 may convert the imagestreams to an uncompressed 30 frames per second video feed comprisingfour differential signals (red, green, blue, and clock signals)according to the Digital Video Interface (DVI) standard.

In additional embodiments, inputs 100 may convert the images streams toother standards, such as a High-definition Multimedia Interface (HDMI)standard, a Red-Green-Blue (RGB) Digital standard, a Red-Green-Blue(RGB) Analog standard, a DisplayPort standard, a National TelevisionSystem Committee (NTSC) standard, a Phase Alternate Line (PAL) standard,Serial Digital Interface (SDI) standard. Accordingly, variousembodiments may have inputs 100 further configured with a variety ofconnector types, thereby enabling system 98 to accept image streamsbased on different protocols, formats, and standards. For example, someof those connector types include Digital Video Interface (DVI),High-definition Multimedia Interface (HDMI), Bayonet Neill-Concelman(BNC), fiber optic, DisplayPort, Universal Serial Bus (USB), andFirewire (IEEE1394).

Continuing with the illustrated embodiment, input modules 100 arecoupled to a switch fabric 101 that allows the converted image streamsto be routed to any of the output modules 103 or function modules 102.For example, switch fabric 101 may comprise a crossbar switch orcrossover switch matrix. The embodiment further comprises a controlmodule 104 in communication with a graphical user interface (GUI) 106and the switch fabric 101. In this embodiment, the control module 104 isin electrical communication with the switch fabric 101 to allow therouting configuration of the switch fabric 101 to be controlled. The GUI106 interfaces with the control module 104 to allow the system user tocontrol which image streams are subject to which image processingfunctions and where those image streams are eventually displayed. Forexample, the GUI 106 might comprise a touch screen interface displayinga graphical representation of the system that is configured to allow auser to control the image stream routing and processing through tactilecommands. In this example, the control module 104 may interface with theGUI 106 and translate the graphical manipulations performed at the GUI106 to the corresponding electrical signals necessary to reconfigure theswitch fabric 101 to conform to the user's desired routing andprocessing configuration.

In this embodiment, the switch fabric 101 is in further communicationwith a plurality of functional image processing modules 102 and aplurality of output modules 103. A functional image processing module102 is configured to receive one or more image streams via the switchfabric 101, to perform a predetermined function on the received imagestreams, and output the results as one or more image streams back intothe switch fabric 101. In some embodiments, the output of the imageprocessing modules 102 is configured to be compatible with the output ofinput modules 100. Accordingly, image streams that are output from afirst function 102 can be utilized as input streams for a secondfunction 102, and so on. For example, live video from a UAV can becombined with a synthetic aperture radar (SAR) data stream for use inde-hazing the live video stream and a de-hazed live video stream may beprovided to an edge detecting functional module to provide a de-hazedand edge detected video stream to a display 105.

Output modules 103 are configured to receive the data streams from theswitch fabric 101 and to format them for display on a plurality ofdisplay devices 105. For example, one of the output modules 103 mayreceive an uncompressed video stream from the switch fabric 101 andformat it to be compressed and encoded for transmission to a displaydevice 105 situated in a remotely located headquarters. As anotherexample, the GUI 106 may be integrated into one or more of the displaydevices 105 such that use of the control module 104 comprisesmanipulating an image displayed on the display device 105 to performsensor fusion. For example, a user may use a touch display to drag aninfrared video stream onto a SAR data stream combining the infraredvideo stream with the SAR data stream through a sensor fusion algorithm.The GUI 106 then causes the control module 104 to route the infraredvideo stream and the SAR data stream onto a fusion functional module 102to display a combined infrared/SAR video stream on the display device105.

It should be noted that, in addition to military applications, thevarious systems and methods in accordance with the present invention arealso particularly useful in the medical field. Specifically, variousembodiments of the invention can be utilized with medical imagingdevices, where image streams provided by medical imaging devices can beprocessed or enhanced for analysis purposes. For example, an endoscopemay be used in conjunction with a system in accordance with oneembodiment, whereby the system receives an image stream from theendoscope, the system maps the image stream to an image processingmodule that applies light optimizing functions to the image stream, andthe system outputs the results from the image processing module to anoutput module for display. By doing so, the image stream from theendoscope is enhanced for polyp detection. The light optimizing functionmay adjust a specific color channel (e.g., red, green, blue) for theimage stream, or may adjust the color within a predetermined wavelengthrange for the image stream (e.g., a predetermined range centered around10% of the center wavelength). In some embodiments, a graphical userinterface may be displayed on the output device in conjunction with theoutput image stream to enable a system user to modify various parametersof the image processing module. For instance, a user may be able toselect a color channel for the displayed image stream and be able tomodify the color channel according to user preferences. For example, auser may desire to modify a red channel of the color stream to increasethe red channel contrast and modify the red channel tint.

FIG. 2B illustrates an exemplary video input module according to anembodiment of the invention. As described herein, data sources used invarious embodiments of the invention may be configured using differentprotocols and formats. For example, an external device 110, such as acomputer, may be configured to wirelessly receive and decode radiotransmitted video signals and provide them in an analog format accordingto the DVI-I protocol. The analog signal is transmitted to theconversion module 111 for conversion to a digital standard, such asTMDS. The digitally converted signal is then provided to the switchfabric 101, as described herein. A control module 112 is provided toimplement system control functions. For example, the control module 112may be configured to monitor the conversion module 111 for reporting tothe system wide control module, such as module 104 described withrespect to FIG. 2A. The control module 112 may be further configured tointerface with the external devices, for example through a digital datacontrol channel or a USB interface. For example, some data sources maybe devices that are controllable, such as a remote sensor configured tocontrollably alter its field of view. Accordingly, in addition toreceiving data from them, some embodiments may be equipped with theability to transfer commands and information to the data sources.

FIG. 2C illustrates a video output module according to an embodiment ofthe invention. As with the input devices, a variety of output devicesmay interface with the system according to various embodiments of theinvention. For example, output devices may comprise radio transmissionlinks, video displays, or recording devices. Accordingly, to interfacewith these various output devices, a video output module may comprise aconversion module 113 configured to convert the signal received from theswitch fabric 101 to an appropriate format for the particular outputdevice 115 coupled to the video output module. A video output controlmodule 114 may also be provided to implement various other functions.For example, control module 114 may monitor the conversion module 113for reporting to the system wide control module. The control module 114may also receive display ID information from the external device 115 andprovide that information to the system wide control module. In someembodiments, the output device 115 may have particular timingrequirements, so the video output module may be further provided with atiming generator module 116 that provides the timing channel for theDVI-D link under the immediate control of the control module 114 andunder general control by the system wide control module. For example,timing generator module 116 may generate a timing channel and provide itto a blending and mixing functional module 102 so that multipleasynchronous image streams may be synchronized for image combination anddisplay on output device 115. Additionally, in systems configured toreceive information from output devices, the control module 114 may beconfigured to relay that information to the system. For example, someembodiments may utilize a USB 2.0 communications protocol to allow theoutput device 115 to transmit data and control information to otherdevices in communication with the system. In a particular embodiment,some of the other devices in communication with the system areconfigured for full duplex communications, so to implement USB-basedcommunications, the control module 114 converts the half-duplex USB 2.0to a full duplex.

FIG. 2D illustrates an exemplary functional image processing moduleaccording to an embodiment of the invention. A function module 120 isprovided to receive one or more input video streams from the switchfabric 101 and to perform image processing algorithms upon them. Theprocessed video streams are output back onto the switch fabric 101 foruse as an input in another function module or transmission to an outputdevice. Additionally, a timing signal, for example as generated by avideo output module described with respect to FIG. 2C, may be receivedby the function module 120 for synchronization and timing formatting.For example, such a timing signal may be used for chromakey-based mixingof two incoming video signals. A control module 121 is further providedto implement system control functions. For example, the control module121 may be configured to monitor the operations of the function module120 for reporting to the system wide control module. In someembodiments, the function module 120 may be configured to implement oneof a plurality of image processing algorithms, and the choice ofalgorithm may be controlled by the control module 121 under command fromthe system wide control. Additionally, some function modules may beconfigured to download and receive new image processing algorithms.These new algorithms may be transmitted through the system control linkand programmed into the function module 120 by the control module 121.

FIG. 2E illustrates a general method for image processing according toan embodiment of the invention. Image stream 130 comprises a pluralityof frames 131 transmitted at particular time intervals, such as 30frames per second. In many cases, image stream 130 is unlikely to varyconsiderably from frame to frame, i.e. frame b is likely to be verysimilar to frame a, frame c will likely be very similar to frame b, andso on. Accordingly, latency may be avoided or reduced by applying theresults of a processing function on a given frame to a subsequent frame.For example, in FIG. 2E, the function 133 is computed for each frame andthe results are applied to the next frame, i.e. F(a), the results of thefunction applied to frame a, are applied to frame b; F(b) is applied toframe c, and so on. In some embodiments, this method may be employedwhen the time it takes function module 120 to process a single imageframe may be less than or equal to the time that image frame isdisplayed. In other embodiments, the results of a function performed ona frame may be applied to a plurality of following frames, or a framethat follows after a number of intervening frames. For example, theresults of the function might be applied to the following three framesand the results function itself may be computed for every third frame.This method might be employed, for example, when computationalrequirements are larger and frame computations cannot be performed inparallel. As another example, the function may be computed for eachframe as before, but the results of the computation might be applied tothe fourth frame following the argument of the function. In other words,F(a) may be applied to frame d, F(b) may be applied to frame e, and soon. This method might be employed, for example, when computational timeis longer than the time it takes each frame to be displayed butsimultaneous frame computations may be performed. In other embodiments,these methods may be combined or implemented together and some latencymay be tolerated according to environment-specific variables such as theimage stream rate of change, the image stream frame rate, and the imageprocessing function or functions being performed on the image stream.

FIG. 3A illustrates an example of a routing process, where the datastream from data source 99A is routed through functional modules 102A,102B, and 102C in that order, and then combined with the data streamfrom data source 99B to be displayed on display device 105A. FIG. 3Billustrates another example of a routing process, implemented on afabric matrix switch in a system where image streams comprise fourdifferential signal pairs, such as TMDS standard signals. In thisexample, the fabric matrix switch card may comprise a switch fabrichaving 144 addressable inputs (input interfaces) and 144 addressableoutputs (output interfaces). The inputs and outputs may be furtherportioned into 36 addressed input ports and 36 addressed output ports,each port comprising four inputs or outputs corresponding to the fourdifferential signal pairs of the image streams. In FIG. 3B, array 139represents the addressed input ports, array 137 represents the addressedoutput ports, and array 138 represents the mappings from the input portsto the output ports.

In this example, an analog video feed 140 is converted to a digitalstandard and provided to the system on input port 1A. Chroma mask 141comprising latitude and longitude coordinates is provided to the systemon input port 2A. A mission ID image is further provided to the systemon input port 2C. Port 1A is mapped to output ports 4A and 5A. Outputport 4A outputs to module functional 145, which enhances the receivedfeed 140 and provides the enhanced feed to input port 4A. Input port 4Ais mapped to output port 4D, which outputs to functional module 144.Input port 2A is mapped to output port 4C, which also outputs functionalmodule 144. Accordingly, functional module 144 applies chroma key imagecombination to the enhanced feed and the chroma mask to provide anenhanced and coordinate-overlaid feed to input port 4C. Input port 4C ismapped to both output port 6A and output port 5B. Output port 6A iscoupled to a video screen 146, so that the enhanced andcoordinate-overlaid feed is displayed on a video screen 146. Output port5B is coupled to functional module 143 along with output port 5A and asecond instance of output port 5B. Functional module 143 is configuredto scale and combine the received image streams and overlay the missionID 142 to form an image stream comprising the original analog videostream 140 in a reduced scale alongside the enhanced andcoordinate-overlaid image stream, with the mission ID displayed near thetop of the images. This mixed stream is provided to input port 5A, whichis mapped to output port 6D and displayed on video screen 147.

FIG. 3C illustrates an exemplary method for switch fabrics according toan embodiment of the invention. A portion of a first switch fabric 150is illustrated with input array portion 151 and output array portion152, and a portion of a second switch fabric 153 is illustrated withinput array portion 154 and output array portion 155. By electricallycoupling the output portions of the switch fabrics in the mannerillustrated, the two switch fabrics may be combined such that an inputdevice coupled to an input port of array portion 151 can be routed to anoutput of array portion 155. For example, input port 8D of array 151might be mapped to output port X1 of array 152 which is in turn mappedto input port 1A of array 154. Input port 1A of array 154 may then bemapped to any of the output ports of array 155. In some embodiments ofthe invention, such fabric coupling may be employed to achieve arequired number of input or output ports. In other embodiments of theinvention, this method of fabric coupling may be extended arbitrarily,thereby allowing multiple embodiments of the invention to be coupledtogether to allow system scaling.

In further embodiments of the invention, the fabric coupling may beconfigured redundantly to enable failover operation. In yet furtherembodiments, the switching fabric is capable of supporting a variety ofstandard protocols in addition to video including, but in no way limitedto, Ethernet, Serial Attached SCSI (SAS), Serial ATA (SATA), and PCIExpress (PCI-E).

FIGS. 4-8 illustrate a process of configuring an embodiment of theinvention for use in the ground station component of a UAV system. Inthis embodiment, a UAV ground station is equipped with a plurality ofmonitors, control systems, and communication devices to allow a numberof personnel to operate one or more UAVs.

FIG. 4 illustrates an exemplary initial view of a GUI of anon-configured system for installation in a UAV ground station. Asillustrated by FIG. 5, configuration begins by labeling the sources anddestinations for the input modules and output modules, respectively, anddetermining the specific functions of the function modules. Next, asillustrated by FIG. 6, the destinations for the input data streams aredetermined. For example, a weather data stream, such as provided by aweather station or other data source, is configured to be displayed in acommon view display device; and the output of the heads-up display of apilot payload operator station is configured to be displayed on thepilot's heads-up display.

In some embodiments, an exemplary third step illustrated in FIG. 7,allows a system user to define one or more virtual displays, such asoptional picture-in-picture displays, that can be activated duringsystem operation. A user can define one or more display areas (i.e.,virtual displays) within any physical or conceptual display screen,which can then serve as a destination for an image stream that may havebeen processed through an image processing module (e.g., scaled,blended, chroma-keyed). For example, a common view display device may bepartitioned into two or more virtual displays to allow multiple inputstreams to be displayed simultaneously, such as displaying multipleversions of a live image stream derived from different input sources ordifferent image enhancement processes.

Additionally, in some embodiments, the user can define one or more touchsensitive areas, also referred to as virtual buttons 189, on an externaldisplay monitor that can be monitored for touch activity to control thesystem or to emit user programmable data strings to one or more externalcomputers or devices (e.g., data strings might be used to tune a radioor other external device) associated with the display.

In the exemplary fourth step, as illustrated in FIG. 8, a plurality ofdesired system commands are named and defined by selecting routingconfigurations corresponding the desired command. For example, asdescribed with respect to FIG. 6, the PPO-1 HUD data stream is routed tothe pilot HUD display and the system may be configured such that acommand is defined by a route between the source and destination that isintersected by one or more functional modules, such as an edge detectioncommand resulting in the PPO-1 data stream being routed through an edgedetection function prior to display on the pilot HUD.

FIG. 9A illustrates how system and connected device control can becentralized or distributed using exemplary system routing functions. Forexample, a human machine interface (HMI) such as a keyboard and mousemay be coupled to the switch fabric such that it is able to control theimplementation of the predetermined commands and to reroute streamsaccording to the desires of the system user. Multiple HMIs may beprovided and may share control or be given control of a subset of theinput and output devices according to system configuration. Accordingly,an additional step of system configuration may comprise mapping the HMIsbetween the display devices, the source devices, and the functionalimage processing modules.

Additionally, although not shown, in some embodiments, a display devicecan communicate human interface device (HID) touch screen or operatorbutton events back to the system through the output interface providingit with an image source. By doing so, the system eliminates the need foran additional cable and or an external computer to monitor and react tothe human interface device events for purposes of controlling thesystem. Additionally, the system can appropriately route human interfacedevice (HID) events to source computers associated with a particularvirtual display inserted within a display device.

For example, the touch screen events may be detected by the displaydevice and passed back to the system (i.e., the image source) using aDisplay Data Channel (DDC) to control the configuration and routing ofthe system. The Display Data Channel (DDC) is a serial bus incorporatedinto some video monitors, thereby allowing some of those specificallyconfigured display devices to provide their image source withdescriptive data called Extended Device Identification Data (EDID).Though this serial channel is normally used to query the display foridentification or operating attributes, in the current example it isbeing utilized to also relay control information from the display deviceto the system. It should be appreciated that other channels similar tothat of the Display Data Channel incorporated into an image sourceconnection could also be utilized to implement similar functionality inother embodiments.

FIG. 9B illustrates an exemplary method of implementing distributedsystem control functions according to an embodiment of the invention. Inthis embodiment, input port array 180, output port array 183, andmapping table 184 are substantially similar to the corresponding arraysdescribed with respect to FIG. 3B. Additionally, control port arrays 182and 185 are provided to allow reconfigurable control of input devices186 and 187. In this embodiment, input devices 186 and 187 areelectrically coupled to output control port array 185 to allow controlsignals to be transmitted from output control port array 185 to theinput devices 186 and 187. For example, the control signals may betransmitted according to a USB standard and the output control portarray 185 may comprise a plurality of addressable USB ports. In theillustrated embodiment, an input control port array 182 is provided withports corresponding to the output ports of array 183. These ports aremapped 181 to output control ports of output control port array 185. Forexample, display device 188 may comprise a touch screen display with avideo connection coupled to output ports 6B of array 183 and a USBconnection coupled to port 6B of array 182. Port 6B of array 182 ismapped to port 7C2 of array 185, which is in turn electrically coupledto input device 187. Accordingly, touch commands executed on display 188are transmitted to input device 187. As described herein, in embodimentsemploying USB or other full-duplex control protocols, modules may becoupled to the ports of array 182 and array 185 to allow the full-duplexcontrol protocols to be translated into corresponding half duplexsignals for transmission through the system and to be retranslated intothe original full-duplex control signals for transmission to the inputdevices.

FIG. 10 illustrates an environment 200 in which an exemplary system 212is collaboratively switching between an image stream input interface andan image stream output interface according to an embodiment of theinvention. The environment 200 comprises the system 212 forcollaboratively switching between image input stream interfaces andimage output stream interfaces, image stream output devices 202, animage input stream hub 226 (e.g., video hub), and a networkswitch/routing device 220. The system 212 may be implemented such thatselective mapping (by the system 212) can be configured on-the-fly frommultiple points of access simultaneously and interactively with two ormore users (e.g., through a computing device, such a laptop, tablet, orsmartphone).

According to some embodiments, the system 212 comprises a plurality ofimage stream input interfaces, a plurality of image stream outputinterfaces, and a switching matrix. Through each of the plurality ofimage stream input interfaces, the system 212 may receive an image inputstream from an image stream generating device, such as a smartphone 232,a personal digital assistant (PDAs) 234, a tablet 236, a laptop 238, adesktop computer 240, a camera 242, a video camera 230, a radar device,a medical device, a surveillance device (e.g., UAVs 214 or satellites228), or an array of servers 210.

Various image stream generating devices may couple to the image streaminput interfaces through an image input stream hub 226, such as a videohub, which allows the system 212 to be remotely located with respect tothe image stream generated devices to which it is coupled. Use of theimage input stream hub 226 may be beneficial where the image streamoutput device 202 is in a conference room setting, the system 212 islocated in a room outside the conference area (e.g., an equipment room),and various users in the conference room wish to couple their respectivelaptops (or other computing devices) to the system 212 for output to theimage stream output device 202. In such a setting, the image stream hub226 may be placed within the conference area, for example, on aconference table.

Through each image stream output interface, the system 212 may output animage input stream (e.g., from an image stream input or as processed byone or more image processing modules) to an image stream output device,such as a projector 204, a large video display 206 (e.g., a plasma orliquid crystal display), or a multi-display video wall 208, any of whichmay function as a video wall. In certain embodiments, the system 212 maybe implemented and integrated into the image stream output device (e.g.,video monitor, computer monitor, or projector) as one or more integratedcomponents. Such an embodiment would, for example, allow a deviceproviding an image stream (e.g., an image stream generating device) tobe directly connected to the image stream output device and selectivelymapped to virtual displays being shown on the image stream outputdevice. Additionally, for some embodiments, an image stream (e.g., fromthe image stream input interface or an image processor module) may beselectively mapped to a virtual display being outputted through a singleimage stream output interface. It should be understood that a singleimage stream output interface may output one or more virtual displayssimultaneously, each possibly having a different image stream source.

The switching matrix in the system 212 may be in communication with theimage stream input interfaces and the image stream output interfaces,and may be configured to selectively map an image stream input interfaceof the plurality of image stream input interfaces to an image streamoutput interface of the plurality of image stream output interfaces.Additionally, for some embodiments, the switching matrix may selectivelymap image streams, received through the image stream input interfaces,to one or more image processor modules (such as the ones describedherein) before selectively mapping the resulting processed image streamsto the image stream output interfaces.

A collaborative control interface of the system 212, which may beembedded in the system 212 and accessible by two or more userssimultaneously, may permit two or more users to operate the system,configure the system (e.g., configure a selectively mapping by theswitching matrix), or check status of the system. For instance, wherethe collaborative control interface is web-based, a web-basedcollaborative control interface 216 may be simultaneously accessible bytwo or more users through their respective computing devices, and thetwo or more users can respectively issue commands to the system 212 thatmay cause the system 212 to a change operation, change configuration, orprovide a status. By this functionality, two or more users cancollaboratively control the system 212.

Those skilled in the art will appreciate that though the collaborativecontrol interface may be capable of simultaneous access by two or moreusers, and may facilitate control of the system 212 by the two or moreusers, this does not exclude a single user from accessing thecollaborative control interface or from controlling the system 212through the collaborative control interface.

By way of example, consider a “war room” situation where a team of usersis assembled to collaborate on strategy and planning, and each teammember (i.e., user) will bring a laptop containing data, notes, andrelated materials. According to some embodiments, at any point during agroup discussion, a member of the team may desire to display informationfrom his image stream output generating device (e.g., laptop, cellphone, camera, or other video source). In response to the new displayedinformation, another team member may desire to present additional videoinformation from his laptop, or another source such as a live video feedfrom a surveillance camera or the UAV 214. Using the system 212 (or somevariation thereof), a first user may, through the web-basedcollaborative control interface 216 (e.g., through their laptop),configure the system 212 to open up a live chat window (e.g., on theirlaptop) on a video wall (e.g., in a first virtual display of the videowall) to communicate with other meeting participants located in a remotelocation. Concurrently or thereafter, a second user may, through theweb-based collaborative control interface 216 (e.g., through theirlaptop), configure the system 212 to open up slide-show presentation(e.g., from a tablet) on the video wall (e.g., in a second virtualdisplay of the video wall) in another window adjacent the live chatwindow. Effectively, team members can dynamically configure the videoswitch on-the-fly and through the web-based collaborative controlinterface 216 accessed over a network connection (e.g., Ethernet orWiFi). Team members may open, close, resize, or reposition a virtualdisplay, and determine the image stream source for the virtual display.

As described herein, the collaborative control interface of the system212 may be implemented as a web-based interface 216, which may beaccessible by two or more users simultaneously and which is generallyaccessed over a network connection. Accordingly, for some embodiments,the web-based collaborative control interface 216 may be accessed byusers using their respective computing devices (e.g., smartphone 232,personal digital assistant (PDA) 234, table 236, laptop 238, desktopcomputer 240) having a web browser. These computing devices mayestablish a network connection with the system 212 through a networkswitch/router 220 (e.g., wireless router 222, or a network switch 224).Depending on the embodiment, access to the web-based collaborativecontrol interface 216 may be restricted to network connections within anintranet, restricted to network connections through an extranet, or maybe generally open to network connections over the Internet.

As also described herein, the collaborative control interface may enablecollaborative operation of the system 212, configuration of the system212, or status viewing of the system 212, by two or more users havingsimultaneous access to the collaborative control interface. Forparticular embodiments, the operation, configuration and status viewingmay be facilitated using commands 218 that the two or more users issueto the system 212 using the collaborative control interface (e.g.,web-based interface 216). With respect to operation of the system 212,commands may include, for example powering the system 212 on or off, orresetting the system 212. With regard to configuring the system 212,commands may include configuring the selective mapping between one ormore image stream input interfaces, one or more image stream outputinterfaces, and (where applicable) one or more image processing modules,settings for the one or more image processor modules (where applicable),user settings (e.g., user type, username, user password), access controlto the collaborative control interface (e.g., user dependent usercontrol), network settings for the system 212, image stream inputinterface or image stream output interface settings (e.g., resolution,frame rate, format, or protocol), or virtual display settings (e.g.,open, close, resize, or reposition). In regard to status viewing of thesystem 212, commands may include requesting a power status, networksettings, image stream input interface settings, image stream outputinterface settings, collaborative control interface settings, or currentoperation of the system 212 (e.g., which interfaces are currentlyactive, which ones have active virtual displays, or which users areusing the system 212).

FIG. 11 illustrates an exemplary web-based collaborative controlinterface 302 in use in an environment 300 according to an embodiment ofthe invention. The web-based collaborative control interface 302 may beused by the system 212 (which collaboratively switches between an imagestream input interface and an image stream output interface) tofacilitate collaborative operation, configuration, or status viewing ofsystem 212 by two or more users. As shown, the web-based collaborativecontrol interface 302 comprises an embedded web service module 304, acommand service module 308, an embedded co-browser proxy module 310, anda datastore 306.

According to some embodiments, the web-based collaborative controlinterface 302 may provide dynamic configuration capabilities through theembedded web service 304 (e.g., hypertext markup language [HTML]service) that works in conjunction with the embedded co-browser proxymodule 310. The embedded web service 304 may provide users with a userinterface (UI) through a web-based control page that allows users tooperate, control, or request status from the system 212. The embeddedco-browser proxy module 310 may allow multiple users to interact withthe web-based control page and change it concurrently and in acollaborative manner.

To enable the operation, configuration, or status viewing of the system212, the command service module 308 may receive, and subsequentlyperform, commands (relating to operation, configuration, or statusviewing) resulting from user interactions with the web-based controlpage. For some embodiments, the command service module may performcommands in accordance with the order in which the commands are received(e.g., timestamps at which commands are received by the web-basedcollaborative control interface 302), the type of command issued (e.g.,administrative-level vs. non-administrative-level command), or the userissuing the command (e.g., administrative user vs. regular user). Thedatastore 306 may store data that facilitates operation by the embeddedweb service module 304, the command service module 308, or the embeddedco-browser proxy module 310.

In environment 300, users 312 may access and interact with the web-basedcollaborative control interface 302 through their respective webbrowsers 314, 316, and 318 on their respective computing devices (e.g.,laptop, tablet, or desktop computer), which may be in networkcommunication with the web-based collaborative control interface 302.When the users 312 access the web-based collaborative control interface302, each of the users 312 may be respectively presented with aweb-based control page 320, 322 or 324. The web-based control page 320,322, and 324 may enable the users 312 to collaboratively interact withthe web-based collaborative control interface 302, thereby permittingcollaborative control of the system 212 through the web-basedcollaborative control interface 302.

While accessing the web-based control page 320, 322, and 324, the users312 may view the current status of the system 212, operate features ofthe system 212, or configure features of the system 212, in or nearreal-time. To request a status from the system 212, operate a feature ofthe system 212, or configure a feature of the system 212, the users 312may issue one or more commands to the system 212 from their respectiveweb-browsers 314, 316, and 318 (i.e., via web-based control pages 320,322, and 324) to the embedded web service module 304 and the embeddedco-browser proxy module 310. The commands, for example, may configurethe system 212 to open, close, resize, or reposition virtual display inan image stream output device (e.g., display, projector, or video wall)and to determine the source of each virtual display (e.g., the imagestream input interface feeding each virtual display). The commandservice module 308 may subsequently receive and perform those commandsfor the system 212.

Though the web-based control page 320, 322, and 324 presented to each ofthe users 312 may be similar to each other, the web-based control page320, 322, and 324 may differ. For example, the web-based control page320 for user #1 may differ from the other web-based control pages 322and 324 based when user #1 last updated his or her web browser 314, theuser credentials used by user #1 to access the web-based page (e.g.,control page may differ between usernames or user types), or the featureof the web-based control page 320 user #1 is currently accessing. Forinstance, where user #1 is an administrative user, and users #2 and #3are non-administrative users, some of the features shown asavailable/accessible on the web-based control page 320 of user #1 maynot be shown as available/accessible on the web-based control pages 322and 324 of users #2 and #3, respectively.

According to some embodiments, the web-based collaborative controlinterface 302 may provide a web-service (e.g., SOAP services) accessibleto applications or systems external to the system 212, thereby allowingsuch applications or systems to remotely control the system 212 throughweb-service calls. In doing so, the web-based collaborative controlinterface 302 may permit the system 212 to function in concert withheterogeneous systems compatible with the system 212. For example, in aconference room setting, the web-services provided by the web-basedcollaborative control interface 302 may allow the system 212 to functionalong with an overall conference control system that controls lightingin the conference room and/or the deployment of monitors and displays(e.g., image stream output devices to which the system 212 may becoupled).

FIG. 12 illustrates output 400 from an exemplary system forcollaboratively switching according to an embodiment of the invention.In output 400, the exemplary system is presenting multiple virtualdisplays 408, 410, 412, 414, and 416 using a display 406. According tosome embodiments, the position and size of the virtual displays 408,410, 412, 414, and 416 may be determined according to the collaborativecontrol and configuration of the exemplary system by two or more users.

The collaborative control and configuration may also determine thesource image stream input stream of each of the virtual displays 408,410, 412, 414, and 416. For example, the virtual display 408 may beconfigured to receive from a laptop 402, the virtual display 410 may beconfigured to receive from a desktop computer 404, the virtual display412 may be configured to receive from an unmanned aerial vehicle (UAV)418, the virtual display 414 may be configured to receive from asurveillance satellite 420, and the virtual display 416 may beconfigured to receive from a tablet 422.

FIG. 13 illustrates an exemplary method 500 for switching controlbetween a plurality of computer systems according to an embodiment ofthe invention. For some embodiments, the method 500 may determine howtwo or more users may collaboratively operate, configure, or view astatus of the system 212 using commands (e.g., command received via acollaborative control interface).

The method 500 may begin at operation 502, where a first command may bereceived from a first user. At operation 504, a second command may bereceived from a second user a second user. The first and second commandsmay relate to operating, configuring, or viewing a status of the system212, and may have been issued by the first and second users throughtheir respective computing devices (e.g., the first user from his or herlaptop, and the second user from his or her tablet). For someembodiments, the first and second commands may be received via acollaborative control interface of the system 212, such as the web-basedcollaborative control interface 302 of FIG. 3.

At operation 506, the first and second commands are stored in a queue atthe system 212. The order in which commands are stored and arranged inthe queue may depend on the particular embodiment. For instance,according to some embodiments, the first and second command may bestored and arranged in the queue according to the order in which thefirst and second commands are received (e.g., based on a transmit orreceive timestamp), the type of command issued (e.g.,administrative-level vs. non-administrative-level command), the userissuing the command (e.g., administrative user vs. regular user), orsome combination thereof. To facilitate preemption of control byadministrative users, some embodiments may store and arrange commandsfrom administrative users in the queue such that they have priority overcommands from non-administrative users (e.g., store the commands at thebeginning of the queue). Further, some embodiments can be configured torestrict storage of commands to only administrative users (e.g., when anadministrative user desires control of the system 212 to the exclusionof all non-administrative users).

At operation 508, the first and second commands may be performed inaccordance with the queue. For some embodiments, performing of commandsin accordance with the queue may involve performing commands accordingto the order in which they are stored in the queue, or according to thepriority of commands in the queue (e.g., higher priority commands in thequeue are performed first).

At operation 510, an original image stream may be received at an imagestream input interface of the system 212. Subsequently, the originalstream may be selectively mapped to an image stream output interface ofsystem 212. The selective mapping of the original image stream may beaccording to the current configuration of the system 212, which may bedetermined (i.e., influenced) by the first and second commands performedat operation 508. Additionally, the configuration of the system 212 maybe such that the original stream (received at the image stream inputinterface) is selectively mapped through one or more image processingmodules of the system 212, and the resulting processed image stream iseventually selectively mapped to the image stream output interface inplace of the original stream.

FIG. 14 illustrates an exemplary system 616 for switching controlbetween a plurality of computer systems according to an embodiment ofthe invention. As shown, the system 616 is coupled to a display 604 (viaconnection 614), computer systems 622, 624, 626, 628, and 630, akeyboard 620, and a mouse 618. The system 616 is selectively mappingimage streams from the computer systems 622, 624, 626, 628, and 630 tovirtual displays 602, 606, 608, 610, and 612, each of which is beingoutputted to the display 604.

According to some embodiments, the system 616 may comprise one or moreimage stream input interfaces, one or more image stream outputinterfaces, and a switching matrix. Through each image stream inputinterface, the system 616 may receive an image input stream from acomputing device (e.g., computer system 622, 624, 626, 628, or 630) thatis configured to couple with a human machine interface, such as thekeyboard 620 or the mouse 618.

Through each image stream output interface, the system 616 may output animage input stream (e.g., from an image stream input or as processed byone or more image processing modules) to an image stream output device,such as the display 604. For some embodiments, the system 616 mayselectively map an image stream (e.g., from the image stream inputinterface or an image processor module) to a virtual display beingoutputted through a single image stream output interface. It should beunderstood that a single image stream output interface may output one ormore virtual displays simultaneously, each possibly having a differentimage stream source.

The system 616 may further comprise one or more central computer inputdevice interfaces and one or more general computer input deviceinterfaces. Each of the central computer input device interfaces may beconfigured to couple to a human machine interface (HMI) devices, whileeach of the general computer input device interfaces may be configuredto couple to an individual computer system. Examples of computer inputdevice interfaces and computer output device outputs may include,without limitation, Universal Serial Bus (USB), FireWire, PS/2, AT,RS-232, and Bluetooth.

The switching matrix in the system 616 may be in communication with theimage stream input interfaces, the image stream output interfaces, thecentral computer input device interfaces, and the general computer inputdevice interfaces, thereby allowing the switching matrix to selectivelymap an image stream input interface to an image stream output interfaceof the plurality of image stream output interface and map a centralcomputer input device interface to a general computer input deviceinterface. For some embodiments, the switching matrix may selectivelymap image streams, received through the image stream input interfaces,to one or more image processor modules (such as the ones describedherein) before selectively mapping the resulting processed image streamsto the image stream output interfaces.

For certain embodiments, the central computer input device interfacesmay be grouped into one or more control input groups, and the generalcomputer input device interfaces may be grouped into one or more controloutput groups, such that thee switching matrix may selectively map acontrol input group may to an control output group by the system 616. Inthis way, each control input group may be coupled to a human machineinterface (HMI) pairing (e.g., keyboard-mouse pairing), each controloutput group may be coupled to an individual computer system, and theswitching matrix can selectively map control information from the HMIpairing to the individual computer system that a computer operator(i.e., user) desires to control.

For some embodiments, the switching matrix of the system 616 may beconfigured to automatically convert control information between acentral computer input device interface and a general computer inputdevice interface when a difference in protocol or interface type existsbetween the two. Further, the automatic conversion between centralcomputer input device interfaces and general computer input deviceinterfaces may include conversion between a control input group and acontrol output group.

In FIG. 14, the connection 614 may couple the display 604 to an imagestream output interface of the system 616. The keyboard 620 and themouse 618 may be coupled to a first control input group of the system616, where the first control input group comprises at least two centralcomputer input device control interfaces (e.g., the keyboard 620 iscoupled to a USB interface of a first control input group, and the mouse618 is coupled to a Bluetooth interface of the first control inputgroup). The output image stream output interface (e.g., VGA, DVI, HDMI,or DisplayPort) of each of computer systems 622, 624, 626, 628, and 630may be coupled to an individual image stream input interface of thesystem 616, and the computer input device interfaces of each of computersystems 622, 624, 626, 628, and 630 may be coupled to an individualcontrol output group of the system 616. The individual control outputgroup may comprise at least two central computer input device controlinterfaces (e.g., a USB interface of the computer system 622 is coupledto a USB interface of a first control output group, and a mouse PS/2interface of the computer system 622 is coupled to a mouse PS/2interface of the first control output group).

The display 604 is just one example of an image stream output devicethat may be coupled to the system 616. Other suitable image streamoutput devices may include, without limitation, a projector or amulti-display configuration. Generally, the display 604 may be one thatis appropriate to function as a video wall (e.g., in a Network OperationCenter, or Emergency Operation Center). For certain embodiments, thesystem 616 may be integrated into the display 604 as one or morecomponents. Where the system 616 is integrated into the display 604, atleast one image stream output interface of the system 616 may beinternally coupled to the display 604. Additionally, where applicable,other image stream output interfaces of the system 616 may be providedas image stream output interfaces of the display 604.

As represented by the dashed lines of FIG. 14, the system 616 isselectively mapping the image stream A from the computer system 622 tothe virtual display 602, the image stream B from the computer system 624to the virtual display 606, the image stream C from the computer system626 to the virtual display 612, the image stream D from the computersystem 628 to the virtual display 610, and the image stream E from thecomputer system 630 to the virtual display 608.

Based on the actions of a computer operator (e.g., with respect to thekeyboard 620 or the mouse 618, or with respect to the system 616), thesystem 616 may selectively map control by the keyboard 620, the mouse618, or both, to a single one of the computer systems 622, 624, 626,628, and 630. When subsequent actions by the computer operatorconstitute a request to the system 616 to switch control (i.e., switchcontrol by the keyboard 620 and the mouse 618 to another computersystem), the system 616 may selectively map control by the keyboard 620,the mouse 618, or both to another one of the computer systems 622, 624,626, 628, and 630. As noted herein, the switching matrix of the system616 may this selective mapping by routing the central computer inputdevice interface coupled to the keyboard 620 or the mouse 618, to thegeneral computer input device interfaces that corresponds to one of thecomputer system 622, 624, 626, 628, or 630.

Exemplary actions by the computer operator that may constitute a requestto switch control between computer systems may include, withoutlimitation, predetermined keystrokes (e.g., on the keyboard 620),predetermined movements of a mouse (e.g., the mouse 618), or a directselection request to the system 616 (e.g., via a control panel coupledto the system 616). Further examples of actions may include tracking themovement and/or positioning of an input cursor (e.g., a keyboard promptor a mouse pointer) that is shown on an image stream output device(e.g., the display 604) and controlled by a computer input device (e.g.,controlled by the keyboard 620 or the mouse 618).

In instances where one or more virtual displays is being outputted to animage stream output device, and where the image stream of at least oneof the virtual displays is being sourced from a given computer system,some embodiments may track the movements and/or positioning of an inputcursor with respect to the virtual displays to determine when control byan computer input device (that is controlling the input cursor) shouldswitch to the given computer system. For some embodiments, movementand/or positioning tracking of the input cursor may be with respect todisplay regions defined relative to the virtual displays. As such, forsome embodiments, display region may be defined with boundaries thatcoincide with or exceed the boundaries of a given virtual display.

In the case of FIG. 14, where multiple virtual displays 602, 606, 608,610, and 612 are being outputted to the display 604 and the computersystems 622, 624, 626, 628, and 630 are providing image streams to thevirtual displays, the system 616 may switch control by the keyboard 620or the mouse 618 based on the movement and/or positioning of an inputcursor that is being controlled by either the keyboard 620 (i.e.,keyboard cursor) or the mouse 618 (i.e., mouse cursor). For instance,when a mouse pointer controlled by the mouse 618 is positioned (on thedisplay 604) within a display region designated for the computer system622 (e.g., the virtual display 602), the system 616 may switch controlof the computer system 622 to the mouse 618, the keyboard 620, or both.Subsequently, when the mouse pointer controlled by the mouse 618 ispositioned (on the display 604) within a display region designated forthe computer system 626 (e.g., the virtual display 612), the system 616may switch control of the computer system 622 to the mouse 618, thekeyboard 620, or both.

As noted herein, according to some embodiments, the display regionsdesignated for the computer system 622 and the computer system 626 maybe respectively defined with boundaries that coincide with or exceed theboundaries of the virtual display 602 being sourced by the computersystem 622.

Under certain embodiments, the computer system being controlled by agiven computer input device may be independently determined based on themovement and/or positioning of the input cursor being controlled by thegiven input device. For instance, in the case of FIG. 14, control of agiven computer system 622, 624, 626, 628, or 630 by the mouse 618 may bedetermined by the system 616 according to the movement and/orpositioning of a mouse pointer being controlled by the mouse 618,independent of the movement and/or position of a keyboard prompt beingcontrolled by the keyboard 620.

Under some embodiments, the computer system being controlled by a givencomputer input device may be tied to a group of computer input devices,such that the movement and/or positioning of any input cursor beingcontrolled by the group can determine which computer system the group ofcomputer input devices is controlling. Based on some embodimentsdescribed herein, computer input devices may be grouped togetheraccording to control input groups of the system 616 (each comprising agrouped set of central computer input device interfaces). For example,if the keyboard 620 and the mouse 618 are coupled to central computerinput interfaces belonged to a common control input group (e.g., firstcomputer input group), control of a given computer system 622, 624, 626,628, or 630 by the keyboard 620 and the mouse 618 may be determined bythe system 616 according to the movement and/or positioning of a mousepointer being controlled by the mouse 618 or the movement and/orposition of a keyboard prompt being controlled by the keyboard 620.

FIG. 15 illustrates an exemplary control interface 700 for switchingcontrol between a plurality of computer systems according to anembodiment of the invention. Through control interfaces such as thecontrol interface 700, a computer operator may be able to define virtualdisplays (referred to as “PIP” in FIG. 15), define behavior of virtualdisplays (e.g., based on computer input device actions), define displayregions for particular virtual displays, or define conditions underwhich control by one or more computer input devices should switchbetween computer systems.

In the particular case of FIG. 15, the control interface 700 may permitthe computer operator switch between the settings of virtual displays702. Once a the settings for a given virtual display is selected, thecontrol interface 700 may permit a computer operator to define streamsources 714 for the given virtual display, define settings 704 for whenthe given virtual display should be iconized, define settings 706 whenthe given virtual display should be expanded, define the location 708 ofthe given virtual display when expanded, and define the location 710 ofthe given virtual display when iconized. To further enable switchingcontrol of computer systems by computer input devices, the controlinterface 700 may allow the computer operator to define coordinates 712of display regions for the mouse and for the keyboard.

FIG. 16 illustrates exemplary image stream and computer input deviceinterfaces 800 of an exemplary system according to an embodiment of theinvention. According to some embodiments, image stream and computerinput device interfaces 800 may comprise a set of (computer)operator-side interfaces 802 and a set of computer side interfaces 816.Generally, the set of (computer) operator-side interfaces 802 may beconfigured to be coupled to computer input devices that feed computerinput device information into the exemplary system (e.g., the system616) and coupled to image stream output devices (e.g., the display 604)that receive image stream output form the exemplary system. The set ofset of computer side interfaces 816, on the other hand, may beconfigured to couple to the computer input device interfaces and imagestream output interfaces of a computer system to be used with theexemplary system.

In accordance with some embodiments, the set of (computer) operator-sideinterfaces 802 may comprise a control input group 804 that includescentral computer input device interfaces, and one or more image streamoutput interfaces 806. The central computer input device interfaces ofthe control input group 804 may be configured to be coupled to thecomputer input devices (e.g., the keyboard 620 and the mouse 618) that acomputer operator uses to control one or more computer systems throughthe exemplary system. The image stream output interfaces 806 may beconfigured to couple to one or more image stream output devices (e.g.,the display 604) used to output image streams from the one or morecomputer systems under the control of the exemplary system. For certainembodiments, the set of operator-side interfaces 802 may include anexternal signal interface 808 for providing one or more image streamoutput devices with control information from the exemplary system.

In various embodiments, the set of set of computer side interfaces 816may comprise interface groupings 810 where each grouping is configuredto be coupled to an individual computer system that is to be controlledthrough the exemplary system. Depending on the embodiment, eachinterface grouping 810 may comprise a control output group 812 thatincludes general computer input device interfaces, and one or more imagestream input interfaces 814. The general computer input deviceinterfaces may be configured to couple to the corresponding computerinput device interfaces of a computer system (e.g., computer system 622,624, 626, 628, or 630) to be controlled through the exemplary system.Likewise, the image stream input interfaces 814 may be configured tocouple to the corresponding computer input device interfaces of acomputer system (e.g., computer system 622, 624, 626, 628, or 630) to becontrolled through the exemplary system.

FIG. 17 illustrates an exemplary method 900 for switching controlbetween a plurality of computer systems according to an embodiment ofthe invention. For some embodiments, the method 900 may determine howthe system 616 selectively maps image streams and control informationfrom computer input devices.

The method 900 may begin at operation 902, where a first image streammay be received at a first image stream input interface of the system616. Some time thereafter, the first image stream is selectively mappedto a first virtual display (e.g., the virtual display 602) beingoutputted through a central image stream output interface at step 904.The image stream output interface may be coupled to an image streamoutput device (e.g., such as the display 604) that will display thefirst virtual display.

At step 906, a second image stream may be received at a second imagestream input interface of the system 616. Eventually, at step 908, thesecond image stream is selectively mapped to a second virtual display(e.g., the virtual display 606) being outputted through a central imagestream output interface. For some embodiments, the first and secondvirtual displays may be outputted through the same central image streamoutput interface and, thus, are being displayed on the same image streamoutput device coupled to the central image stream output.

At step 910, input information may be received at a central computerinput device interface. For some embodiments, the central computer inputinterface may be coupled to a computer input device (e.g., the keyboard620 or the mouse 618) configured to control one or more computer systemscoupled to the system 616.

At step 912, the input information from the central computer inputdevice may be selectively mapped to either the first computer inputdevice interface (e.g., coupled to the computer system 602) or thesecond computer input device interface (e.g., coupled to the computersystem 624). In accordance with certain embodiments described herein,the selective mapping of the input information may be determined basedon action by a computer operator, particularly with respect to thecomputer input device coupled to the central computer input deviceinterface. For example, the selective mapping may be determined based onthe movement and/or position of an input cursor that is controlled by acomputer input device coupled to the central computer input deviceinterface. For determination purposes, the movement and/or position ofthe input cursor may be judged relative to a display region associatedwith the first virtual display or the second virtual display. It will beunderstood that subsequent actions by the computer operator may resultin the system 616 selective mapping of input information changing fromcomputer input device interface to the other.

For some embodiments in accordance with this invention, the varioussystems described herein are implemented into a display, such as a videomonitor, computer monitor, or projector system, as one or moreintegrated components. Such an embodiment would, for example, allow adevice providing an image stream to directly connect to the display andhave various image processing functions applied to the image streambefore being shown on the display output. Additionally, for someembodiments where the system is integrated into a display, a mechanismfor controlling the system may be provided either through inputs andfeatures integrated into the display (e.g., on screen menu and buttonson the front of the display), or through a separate control mechanismdetached from the display, such as touch panel. Optionally, where thiscontrol mechanism is separate and detached from the display, the controlmechanism may be an overlay configured to be placed over the displayoutput. In some embodiments, the control mechanism is directly connectedto the system and system modules to avoid relaying system controlsthrough the output device. For example, if system outputs are to a videoinput of a computer, the control mechanism is connected directly to thesystem, so that system control commands are not directed through thecomputer.

Although some embodiments of the invention have been described ascomprising circuit cards, either horizontal or vertically oriented, andmaking up a switched fabric, alternative embodiments may be employed.For example, FIG. 9C illustrates such an alternative embodimentutilizing vertical 5.25″ circuit cards.

As used herein, the term module might describe a given unit offunctionality that can be performed in accordance with one or moreembodiments of the present invention. As used herein, a module might beimplemented utilizing any form of hardware, software, or a combinationthereof. For example, one or more processors, controllers, ASICs, PLAs,PALs, CPLDs, FPGAs, logical components, software routines or othermechanisms might be implemented to make up a module. In implementation,the various modules described herein might be implemented as discretemodules or the functions and features described can be shared in part orin total among one or more modules. In other words, as would be apparentto one of ordinary skill in the art after reading this description, thevarious features and functionality described herein may be implementedin any given application and can be implemented in one or more separateor shared modules in various combinations and permutations. Even thoughvarious features or elements of functionality may be individuallydescribed or claimed as separate modules, one of ordinary skill in theart will understand that these features and functionality can be sharedamong one or more common software and hardware elements, and suchdescription shall not require or imply that separate hardware orsoftware components are used to implement such features orfunctionality.

Where components or modules of the invention are implemented in whole orin part using software, in one embodiment, these software elements canbe implemented to operate with a computing or processing module capableof carrying out the functionality described with respect thereto. Onesuch exemplary computing module is shown in FIG. 18. Various embodimentsare described in terms of this exemplary computing module 1000. Afterreading this description, it will become apparent to a person skilled inthe relevant art how to implement the invention using other computingmodules or architectures.

Referring now to FIG. 18, computing module 1000 may represent, forexample, computing or processing capabilities found within desktop,laptop and notebook computers; hand-held computing devices (FDA's, smartphones, cell phones, palmtops, etc.); mainframes, supercomputers,workstations or servers; or any other type of special-purpose orgeneral-purpose computing devices as may be desirable or appropriate fora given application or environment. Computing module 1000 might alsorepresent computing capabilities embedded within or otherwise availableto a given device. For example, a computing module might be found inother electronic devices such as, for example, digital cameras,navigation systems, cellular telephones, portable computing devices,modems, routers, WAPs, terminals and other electronic devices that mightinclude some form of processing capability.

Computing module 1000 might include, for example, one or moreprocessors, controllers, control modules, or other processing devices,such as a processor 1004. Processor 1004 might be implemented using ageneral-purpose or special-purpose processing engine such as, forexample, a microprocessor, controller, or other control logic. In theillustrated example, processor 1004 is connected to a bus 1002, althoughany communication medium can be used to facilitate interaction withother components of computing module 1000 or to communicate externally.

Computing module 1000 might also include one or more memory modules,simply referred to herein as main memory 1008. For example, preferablyrandom access memory (RAM) or other dynamic memory, might be used forstoring information and instructions to be executed by processor 1004.Main memory 1008 might also be used for storing temporary variables orother intermediate information during execution of instructions to beexecuted by processor 1004. Computing module 1000 might likewise includea read only memory (“ROM”) or other static storage device coupled to bus1002 for storing static information and instructions for processor 1004.

The computing module 1000 might also include one or more various formsof information storage mechanism 1010, which might include, for example,a media drive 1012 and a storage unit interface 1020. The media drive1012 might include a drive or other mechanism to support fixed orremovable storage media 1014. For example, a hard disk drive, a floppydisk drive, a magnetic tape drive, an optical disk drive, a CD or DVDdrive (R or RW), or other removable or fixed media drive might beprovided. Accordingly, storage media 1014 might include, for example, ahard disk, a floppy disk, magnetic tape, cartridge, optical disk, a CDor DVD, or other fixed or removable medium that is read by, written toor accessed by media drive 1012. As these examples illustrate, thestorage media 1014 can include a computer usable storage medium havingstored therein computer software or data.

In alternative embodiments, information storage mechanism 1010 mightinclude other similar instrumentalities for allowing computer programsor other instructions or data to be loaded into computing module 1000.Such instrumentalities might include, for example, a fixed or removablestorage unit 1022 and an interface 1020. Examples of such storage units1022 and interfaces 1020 can include a program cartridge and cartridgeinterface, a removable memory (for example, a flash memory or otherremovable memory module) and memory slot, a PCMCIA slot and card, andother fixed or removable storage units 1022 and interfaces 1020 thatallow software and data to be transferred from the storage unit 1022 tocomputing module 1000.

Computing module 1000 might also include a communications interface1024. Communications interface 1024 might be used to allow software anddata to be transferred between computing module 1000 and externaldevices. Examples of communications interface 1024 might include a modemor softmodem, a network interface (such as an Ethernet, networkinterface card, WiMedia, IEEE 802.XX or other interface), acommunications port (such as for example, a USB port, IR port, RS232port Bluetooth® interface, or other port), or other communicationsinterface. Software and data transferred via communications interface1024 might typically be carried on signals, which can be electronic,electromagnetic (which includes optical) or other signals capable ofbeing exchanged by a given communications interface 1024. These signalsmight be provided to communications interface 1024 via a channel 1028.This channel 1028 might carry signals and might be implemented using awired or wireless communication medium. Some examples of a channel mightinclude a phone line, a cellular link, an RF link, an optical link, anetwork interface, a local or wide area network, and other wired orwireless communications channels.

In this document, the terms “computer program medium” and “computerusable medium” are used to generally refer to media such as, forexample, memory 1008, storage unit 1020, media 1014, and channel 1028.These and other various forms of computer program media or computerusable media may be involved in carrying one or more sequences of one ormore instructions to a processing device for execution. Suchinstructions embodied on the medium, are generally referred to as“computer program code” or a “computer program product” (which may begrouped in the form of computer programs or other groupings). Whenexecuted, such instructions might enable the computing module 1000 toperform features or functions of the present invention as discussedherein.

While various embodiments of the present invention have been describedabove, it should be understood that they have been presented by way ofexample only, and not of limitation. Likewise, the various diagrams maydepict an exemplary architectural or other configuration for theinvention, which is done to aid in understanding the features andfunctionality that can be included in the invention. The invention isnot restricted to the illustrated exemplary architectures orconfigurations, but the desired features can be implemented using avariety of alternative architectures and configurations. Indeed, it willbe apparent to one of ordinary skill in the art how alternativefunctional, logical or physical partitioning and configurations can beimplemented to implement the desired features of the present invention.Also, a multitude of different constituent module names other than thosedepicted herein can be applied to the various partitions. Additionally,with regard to flow diagrams, operational descriptions and methodclaims, the order in which the steps are presented herein shall notmandate that various embodiments be implemented to perform the recitedfunctionality in the same order unless the context dictates otherwise.

Although the invention is described above in terms of various exemplaryembodiments and implementations, it should be understood that thevarious features, aspects and functionality described in one or more ofthe individual embodiments are not limited in their applicability to theparticular embodiment with which they are described, but instead can beapplied, alone or in various combinations, to one or more of the otherembodiments of the invention, whether or not such embodiments aredescribed and whether or not such features are presented as being a partof a described embodiment. Thus, the breadth and scope of the presentinvention should not be limited by any of the above-described exemplaryembodiments.

Terms and phrases used in this document, and variations thereof, unlessotherwise expressly stated, should be construed as open ended as opposedto limiting. As examples of the foregoing: the term “including” shouldbe read as meaning “including, without limitation” or the like; the term“example” is used to provide exemplary instances of the item indiscussion, not an exhaustive or limiting list thereof; the terms “a” or“an” should be read as meaning “at least one,” “one or more” or thelike; and adjectives such as “conventional,” “traditional,” “normal,”“standard,” “known” and terms of similar meaning should not be construedas limiting the item described to a given time period or to an itemavailable as of a given time, but instead should be read to encompassconventional, traditional, normal, or standard technologies that may beavailable or known now or at any time in the future. Likewise, wherethis document refers to technologies that would be apparent or known toone of ordinary skill in the art, such technologies encompass thoseapparent or known to the skilled artisan now or at any time in thefuture.

The presence of broadening words and phrases such as “one or more,” “atleast,” “but not limited to” or other like phrases in some instancesshall not be read to mean that the narrower case is intended or requiredin instances where such broadening phrases may be absent. The use of theterm “module” does not imply that the components or functionalitydescribed or claimed as part of the module are all configured in acommon package. Indeed, any or all of the various components of amodule, whether control logic or other components, can be combined in asingle package or separately maintained and can further be distributedin multiple groupings or packages or across multiple locations.

Additionally, the various embodiments set forth herein are described interms of exemplary block diagrams, flow charts and other illustrations.As will become apparent to one of ordinary skill in the art afterreading this document, the illustrated embodiments and their variousalternatives can be implemented without confinement to the illustratedexamples. For example, block diagrams and their accompanying descriptionshould not be construed as mandating a particular architecture orconfiguration.

1-25. (canceled)
 26. A method for collaboratively switching an imagestream input to an image stream output, comprising: receiving a firstcommand from a first user by way of a collaborative control interfaceaccessible by two or more users simultaneously, wherein the firstcommand relates to configuring a switching matrix in communication witha plurality of image stream input interfaces and an image stream outputinterface, and wherein the switching matrix is configured to selectivelymap an image stream input interface of the plurality of image streaminput interfaces to the image stream output interface; receiving asecond command from a second user by way of the collaborative controlinterface, wherein the second command relates to configuring theswitching matrix; storing the first command and the second command in aqueue; performing the first command and the second command in accordancewith the queue, thereby configuring the switching matrix in accordancewith the first command and the second command; receiving an originalimage stream from the image stream input interface of the plurality ofimage stream input interfaces; and using the switching matrix toselectively map the original image stream to the image stream outputinterface.
 27. The method of claim 26, wherein using the switchingmatrix to selectively map the original image stream to the image streamoutput interface comprises: using the switching matrix to selectivelymap the original image stream to an image processing module, wherein theimage processing module is configured to accept the original imagestream from the image stream input interface of the plurality of imagestream input interfaces, apply an image processing function to theoriginal image stream, and output a processed image stream; and usingthe switching matrix to selectively map the processed image stream tothe image stream output interface or another image processing module.28. The method of claim 27, wherein the image processing functioncomprises an image stream mix function, an image stream scale function,an image stream blend function, an image stream encoding algorithm, oran image stream enhancement function.
 29. The method of claim 28,wherein the image stream enhancement function comprises a de-hazefunction, a de-blur function, a shadow function, a dawn-dusk function, afusion function, a stabilization function, a thermal turbulencefunction, an equalization function, an edge detection function, a rainand fog function, or a light optimizing function.
 30. The method ofclaim 28, wherein the image stream enhancement function performed by theimage processing module comprises an auto-focus function capable ofadjusting focus of an image stream capture device that is coupled to theimage stream input interface and producing the image stream.
 31. Themethod of claim 26, wherein storing the first command and the secondcommand in the queue is based on an order in which the first command andthe second command are received, based on a first priority of the firstcommand and a second priority of the second command, or based on a firstuser type of the first user and a second user type of the second user.32. The method of claim 26, wherein the collaborative control interfacecomprises a web-based interface that enables configuration of theswitching matrix, and wherein the web-based interface is accessible bythe users by way of a network connection with the system.
 33. The methodof claim 26, wherein the first user is an administrative user, whereinthe second user is a non-administrative user, and wherein performing thefirst command and the second command in accordance with the queuecomprises performing the first command from the administrative userbefore the second command from the non-administrative user.
 34. Themethod of claim 26, wherein the switching matrix outputs the originalimage stream to a virtual display, in a mapped image stream, beingoutputted to the image stream output interface.
 35. The method of claim34, wherein the first command or the second command includes creatingthe virtual display, closing the virtual display, resizing the virtualdisplay, repositioning the virtual display within the mapped imagestream being outputted to the image stream output interface, or movingthe virtual display to another mapped image stream being outputted toanother image stream output interface. 36-44. (canceled)
 45. A methodfor switching control between a plurality of computer systems,comprising: receiving a first image stream at a first image stream inputinterface configured to couple with a first computer system of theplurality of computer systems; using a switching matrix to map the firstimage stream from the first image stream input interface to a firstvirtual display being outputted through a central image stream outputinterface; receiving a second image stream at a second image streaminput interface configured to couple with a second computer system ofthe plurality of computer systems; using the switching matrix to map thesecond image stream from the second image stream input interface to asecond virtual display being outputted through the central image streamoutput interface; receiving input information from a central computerinput device interface; and using the switching matrix to selectivelymap the control information from the central computer input deviceinterface to a first computer input device interface or a secondcomputer input device interface based on a location of an input cursorwith respect to a first display region or a second display region,wherein the first display region and the second display region are beingoutputted through the central image stream output interface, wherein thefirst computer input device interface is configured to couple with thefirst computer and the second computer input device interface isconfigured to couple with the second computer system, wherein the firstvirtual display is in the first display region and the second virtualdisplay is the second display region, and wherein the input cursor iscontrolled by way of the central computer input device interface. 46.The method of claim 45, wherein the first virtual display coincides withthe first display region, and wherein the second virtual displaycoincides with the second display region.
 47. The method of claim 45,wherein the input cursor is a mouse being controlled by a mouse coupledto the central computer input device interface.
 48. The method of claim45, wherein the input cursor is a keyboard cursor being controlled by akeyboard coupled to the central computer input device interface.
 49. Themethod of claim 45, wherein selectively mapping the central computerinput device interface to the first computer input interface or thesecond computer input interface is further based on whether the inputcursor is entering or exiting the first display region or the seconddisplay region.
 50. The method of claim 45, wherein using the switchingmatrix to map to map the first image stream input interface to the firstvirtual display comprises: mapping the original image stream from thefirst image stream input interface to an image processing moduleconfigured to accept an original image stream from the first imagestream input interface, apply an image processing function to theoriginal image stream, and output a processed image stream; and mappingthe processed image stream from the first image processing module to thefirst virtual display.
 51. The method of claim 50, wherein the imageprocessing function comprises an image stream mix function, an imagestream scale function, an image stream blend function, an image streamencoding algorithm, or an image stream enhancement function.
 52. Themethod of claim 51, wherein the image stream enhancement functioncomprises a de-haze function, a de-blur function, a shadow function, adawn-dusk function, a fusion function, a stabilization function, athermal turbulence function, an equalization function, an edge detectionfunction, a rain and fog function, or a light optimizing function. 53.The method of claim 52, wherein the image stream enhancement functionperformed by the image processing module comprises an auto-focusfunction capable of adjusting focus of an image stream capture devicethat is coupled to the image stream input interface and producing theimage stream.